Evaluating Predictive Models of Student Success: Closing the Methodological Gap
نویسندگان
چکیده
Model evaluation – the process of making inferences about the performance of predictive models – is a critical component of predictive modeling research in learning analytics. In this work, we present an overview of the state-of-the-practice of model evaluation in learning analytics, which overwhelmingly uses only näıve methods for model evaluation or, less commonly, statistical tests which are not appropriate for predictive model evaluation. We then provide an overview of more appropriate methods for model evaluation, presenting both frequentist and a preferred Bayesian method. Finally, we apply three methods – the näıve average commonly used in learning analytics, frequentist null hypothesis significance test (NHST), and hierarchical Bayesian model evaluation – to a large set of MOOC data. We compare 96 different predictive modeling techniques, including different feature sets, statistical modeling algorithms, and tuning hyperparameters for each, using this case study to demonstrate the different experimental conclusions these evaluation techniques provide.
منابع مشابه
Dropout Model Evaluation in MOOCs
The field of learning analytics needs to adopt a more rigorous approach for predictive model evaluation that matches the complex practice of model-building. In this work, we present a procedure to statistically test hypotheses about model performance which goes beyond the state-of-the-practice in the community to analyze both algorithms and feature extraction methods from raw data. We apply thi...
متن کاملStudent Success Prediction in MOOCs
Predictive models of student success in Massive Open Online Courses (MOOCs) are a critical component of effective content personalization and adaptive interventions. In this article we review the state of the art in predictive models of student success in MOOCs and present a dual categorization of MOOC research according to both predictors (features) and prediction (outcomes). We critically sur...
متن کاملA Framework for Reducing the College Success Gap and Promoting Success for All
Policymakers, practitioners, and scholars have directed tremendous attention to the goal of improving “student success.” Based on a review of largely discrete existing bodies of literature, this report proposes an overarching framework that policymakers, practitioners, and researchers can use to develop, implement, and evaluate policies and practices for addressing persistent racial/ethnic and ...
متن کاملPREDICTIVE ANALYTICS FOR STUDENT SUCCESS: Developing Data-Driven Predictive Models of Student Success
متن کامل
An Analysis of a Need-based Student Aid Program for Georgia
There is a large gap in college enrollment by family income, and there is evidence that this gap is growing. Yet the benefits of a college education, both for the individual and society, are significant. Encouraging college education in general and closing that income gap in enrollment is a long standing policy objective for the United States and Georgia. One policy aimed at closing the income ...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
- CoRR
دوره abs/1801.08494 شماره
صفحات -
تاریخ انتشار 2018